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ABSTRACT 

A workbook designed to teach creative attitudes and 
idea-producing techniques, previously used successfully with 
middle-class students, was field tested with 6th and 8th grade inner 
city students divided into training and control groups. Results were 
evaluated by an attitude survey and three subtests from the Torrance 
Battery. The results were partly negative or ambiguous and it was 
impossible to draw conclusions about the effectiveness of the 
workbook in this situation. However, 75% of the students in the 
training group felt that they had benefited from the course and this 
was endorsed by a majority of their teachers. The training materials, 
the low abilities of the students in the classes chosen, the 
teacher® s handling of the materials and class (due to inadequate 
pretraining) , insensitivity of the tests, and oversensitivity of the 
survey,, are all factors that may have affected the results, thus 
making the apportionment of blame impossible. (GS) 
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Problems in Assessing the Effectiveness 
of Creative Thinking* 



Gary A. Davis 
University of Wisconsin 

Today, I wanted to briefly summarize a few of the major 
problems I recently encountered in attempting to field test some 
creativity training materials with 6th and 8th grade low-ability, 
inner-city students. Essentially, my report will be an applied 
version of the theoretical problems described by other members 
of this panel . 

1 will first present a brief overview of the training materials, 
creativity tests, and the experiment, and then describe some of 
the problems associated with each of our dependent creativity 
measures. 

The training materials consisted of a 150-page workbook entitled 
"Thinking Creatively; A Guide to Training Imagination," which we 
designed for use by 6th, 7th, and 8th~grade students. The workbook 
is in the form of a humorous dialogue among four cartoon story 
characters. Mr. 1 is a backyard scientist-inventor type, who leads 
Dudley, Maybelle, and Max (our professional bear) through various 
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in Both Directions," American Educational Research Association, 
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enlightening problem-solving episodes in which Mr. I attempts to 
teach the other three characters various principles and procedures 
of creative problem solving. At the end of each of the ten 
chapters, the important principles are reviewed and exercises are 
presented which allow the students themselves to solve problems 
similar to those solved by the cartoon characters within the 
story line. 

The workbook mainly attempts to teach students a particular 
set of attitudes which are considered to be conducive to creative 
behavior. For example, one of the most important creative attitudes 
is simply an appreciation for unusual, imaginative ideas. The 
program also teaches various "forced combination" techniques for 
systematically producing new combinations of ideas. 

Regarding tests, since the program teaches creative attitudes 
plus techniques for producing ideas , we used a 20-item afct j tude 
survey, along with three subtests from the Torrance Battery (cue 
Product Improvement , Unusual Uses , and Consequences tests) . 

In addition to the attitude scales and the "creativity tests," 
which four control and four experimental classes received, the 
students and teachers in the experimental, training classes were 
asked additional questions. For example, both students and teachers 
were asked if the program was difficult to read or understand. 
Students were asked rather directly, "Do you feel more creative as 
a result of the training?" Teachers also were asked if they thought 
their students benefited from the training experience. 






So much for the materials and tests. Now: What happened? 

Let me begin by presenting a truism relating to experimental 
control and the interp?;etation of results. The types of controls 
and dependent measures used in this frankly difficult, real-world 
ffeld study — the random selection of schools, the clean, f actor ially- 
balanced experimental design, the use of commercially-available, 
standardized tests — these controls allow one to draw conclusions 
pertaining to the materials only w hen the experimental result s ar£ 
clearly positive . That is, if the trained classes had clearly 
performed more creatively than untrained classes, we undoubtedly 
would have concluded that our materials absolutely can increase the 
creative-thinking skills of inner-city students. On the other hand, 
if the results are partly negative or ambiguous, as in fact they 
were in this field test, we found it impossible to draw conclusions 
regarding the effectiveness or ineffectiveness of the training 

* 

materials. 

Imagine, for a moment, that you have analyzed the test results 
of a creativity-training study with below-average IQ, inner-city 
students . 

consider , first, the Torrance Tests of Creativity Thinking. Jhe 
Torrance tests generally showed little difference between the 
trained and untrained classes. More specifically, just two of the four 
experimental classes showed a moderate improvement in Torrance test 
How, is this partly-negative outcome because the traini n g . 
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materials are only slightly effective? Or maybe the training 
materials are not effective for these particular students , who 
were obviously very weak in "basic skills." Average reading 
ability runs 2-3 years below grade level. Or maybe X should blame 
the teachers of the training classes. Actually, since two of 
the four experimental classes showed moderate gains in fluency , 
flexibility , and originality scores, perhaps half of my teachers 
were not adequately trained to effectively use the materials. 

In addition to the truth-obscuring problems with students and 
teachers, we have the Torrance tests themselves. Can we be sure 
that these tests are reliable, valid and, most important of all, 
sensitively accurate in measuring creativity? Hardly! 

Now to the problem of measuring creative attitudes . In a 
sentence: We found fairly strong, positive effects of the training 

but — and here is the problem — these (quote) "creative attitudes" 
could be social desirability responses. That is, the group who 
read the program knew the "right" attitudes to display on the 
questionnaire. The dilemma here is whether to use test items 
similar to the content of the training materials — and increase the 
likelihood of getting positive results— or to use highly dissimilar 
attitude items, thereby decreasing the sensitivity of the test 
instrument . 

As a final dependent measure, we had the testimonies of the 
students and teachers of the experimental classes. About 75% of 
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the trained students agreed that they "felt more creative" as a 

> 

result of the training. Also, two of the four teachers of the 
training classes gave an unqualified "definitely" when asked if 
their students benefited from the creativity training. Another 
teacher gave qualified support (and this is instructive), saying, 
"Yes, at least they benefited as much or more from this experience 
as from any other that we tried." The fourth teacher thought that 
very few of her low-ability, low-originalitv, undisciplinable 
students learned anything from the training. 

In sum, my conclusions related to assessing creativity training 
are these: Again, if the results of a difficult, complex field 

test such at this are partly negative or ambiguous, one can reach 
no conclusions regarding the value of 4 the materials. We don’t 
know whether to blame the training materials themselves, the low 
abilities of our student population, the teacher’s possible mis- 
handling of the materials and the class (due to our not training 

them properly), the Insensitivity of the divergent thinking tests, 

% 

the over sensitivity of attitude surveys, or perhaps other problems 
whose discovery is still pending. 

To end on a positive note, we remain convinced that our 
training program can increase the creative thinking attitudes and 
skills of disadvantaged, inner-city students, just as it did with 
middle-class students in an earlier study. However, I am having 
a devil of a time unambiguously proving it. , 




1 



